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ABSTRACT 



In order to measure the value of student feedback in 
contributing to the refinement of instructional frames in 
computer-assisted instruction (CAI) , a stand-alone CAI frame was 
written around a question designed to elicit a wide variety of 
replies. Half of a real sample of fifty handwritten responses were 
submitted to the frame, with a match score of 96% and a sense score 
of 48%, After revision, the frame attained a match score of 100% and 
a sense score of 88%, on the other half of the test set. Conclusions 
drawn from the study were that (1) for an open question with a large 
number of legal answers and an adult, heterogeneous student set, a 
striking improvement in meaningful intercept can be obtained through 
close analysis of a modest number of live responses; and (2) as a 
corollary, the rate of improvement to be derived from close analysis 
beyond this mcdest number must decline quite rapidly. In the report 
of the study, selected- string-match CAI strategies and the mechanics 
of the CAI n macrcframe M are discussed* Appendices contain supporting 
materials . ( Authcr/MF) 
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STUDENT FEEDBACK AS A TOOL IN 
COMPUTER-ASSISTED INSTRUCTION (CAI) 
FRAME DEVELOPMENT 

by 

Joseph C. Meredith (Research Center for Library and Infor- 
mation Science, Graduate Library School, Indiana University) 
and Douglas Ferguson (Editor, Automation Department, Stan- 
ford University Libraries) 

Background and objective 

This paper represents an attempt to measure the value of 
student feedback in contributing to the refinement of instructional 
frames in Computer-Assisted Instruction (CAI). In our own 
work in CAI for graduate students (specifically for students of 
library science) (i) we have concentrated on the development of 
a « machine tutorial mode » suitable for presentation of con- 
ceptual material to an adult and very mixed student population. 
We must keep in mind students whose sole preparation is a 
liberal arts degree — others who hold degrees in the sciences • — 
and still others with abundant experience as practicing librarians. 
Primary interests range from children's libraries to the upper 
reaches of information theory. Ages range quite evenly through 
the twenties, thirties, and forties. 

In order to be effective on an individual basis, we have felt 
that we must make the machine side of the program as versatile 
as possible, with a very broad capability for dealing with idio- 
syncratic responses. Whatever we do will fall far short of human 
discourse; we cannot pretend to cope with meanings which have 
not been anticipated; but we can at least take advantage of 
observed patterns of speech and thought in order to avoid more 
formal constraints on dialogue than are absolutely necessary. 

(1) Described in the interim report on Project No. 7-1085 (OEG-1-7- 
071085-4286) : An Information processing laboratory for education and research in 
library science : Phase I (M. E. Maron, A. J. Humphrey, and J. C. Meredith, 
DHEW (OE) July, 1969). 
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What about these patterns ? It is sometimes said that pre- 
planned conversational CAI quickly reaches an area of diminished 
return, because of the massive intellectual and clerical effort 
required to prepare even a short sequence (i). This might be 
true if the CAI author were to work in a vacuum, drawing on 
the single resource of his own semantic store. But what if he were 
to demand less of this for-the-most-part intuitive, remembered, 
and general resource and rely more on live results derived from 
preliminary testing ? We know that student feedback is useful 
in revising programmed learning, but we need to know more 
about the extent of this usefulness and to decide when and how 
best to exploit it. 

Titis, quite simply, was the idea behind what we thought would 
be a straightforward exercise in recursive improvement of a 
single CAI frame. If we had had an earlier indication of how 
striking the results would be, we would have structured the test 
more rigorously. Even so, we feel that the controls are adequate 
to justify reasonable confidence. Also, we did take the precaution 
of drafting and documenting Part I of this paper before proceeding 
with the second part of the experiment, which is described in 
Part II. 



Part I 



r 



§ 
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Procedure 

Fifty individuals were polled in writing (Appendix A) with a 
request that they indicate what single response they might have 
given to a certain statement /question, in an on-line situation at 
a student terminal. Individuals were chosen at random from among 
library school students, faculty, and technical and clerical staff of 
the Institute of Library Research, University of California, Ber- 
keley, California. In using subjects from this environment it 
was hoped that we could compensate in part for the lack of 



(i) William R. Uttal et al. represent this view in Generative Computer 
Assisted Instruction (Communication ^243 of the Mental Health Research 
Institute, University of Michigan, January, 1969). 
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instructional preparation leading up to the statement /question. 
At the same time we realized that a residuum of this lack of 
context would need to be accommodated within the frame itself, 
to an uncommon degree. « Success », as we viewed it, would be 
measured as much by the effectiveness of this accommodation 
as by the efficiency of the match and fail processes of the system. 

Respondents were asked to write down their first response to 
the question as worded. No explanation or comment was provided 
at the time. The fifty answers were serially divided into sets of 
twenty-five each to make up an « A » list and a « B » list, and the 
former was submitted to a previously-written program called 
the « A-Frame ». The results are described and analyzed in the 
sections which follow. 

We then revised the A-Frame in the light of what we had 
learned, and tested the effectiveness of this version (the « B- 
Frame ») against the « B » list of anwers. Appendices A through 
I document the steps which were taken before conducting this 
second test. 



The Program 

In designing the A-Frame we assumed programming language 
capabilities corresponding with PILOT (1), the CAI language in 
use in the Institute’s Information Processing Laboratory, with 
minor technical augmentations provided in DISCUS (2). Both 
languages employ selected character string match as the basic 
answer-matching routine. That is, student input of any length 
will be successfully matched in a scanning operation if the elements 
specified in the scan are present. The remainder of the input is 
ignored. 



(1) See R. Karpinski et al., PILOT ; a Conversational Language — User 9 s 
Guide (Office of Information Systems, University of California Medical Center, 
San Francisco, California, December, 1968). 

(2) Developed by Steven S. Silver, ILR staff member, and now implemented 
at both the Berkeley and Los Angeles campuses of the University of California. 
Documentation available from the Institute of Library Research, University of 
California at Los Angeles, California. User’s Manual in Press. 
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A successful match between a scan element and an input element 
(student answer) triggers a unique set of program reactions. 
A match does not signify tightness or wrongness of a response. 
It refers only to the detection of a sought element. 

In practice, the reaction provided for a response which is 
correct from the teacher’s viewpoint will consist of displaying to 
the student a statement to that effect, then making a record of the 
fact • — for scoring and control purposes — and then exiting to 
the next frame. The reaction provided for incorrect matched 
responses may consist of a critical or cueing display followed by a 
return to the question. Or it may be followed by a critical statement 
incorporating the correct answer and then exiting to the next 
frame. The same scoring and control tallies can accompany this 
operation. 

Lastly, the program reaction on failure to match any of the input 
is usually the display of a succession of two or three cueing re- 
sponses ending with one which gives the correct answer. The 
student is then passed to the next frame, or in rare cases he may 
be advised to sign off and seek the help of an instructor. Fail- 
match tallies are also recorded for control and scoring purposes. 
Naturally, dependence on fail routines must be kept to a minimum 
if meaningful instructional dialogue is to be attained. 



The Frame 

The frame, as such, is primarily an instructional concept. It 
may be programmed in a variety of ways, but usually it is com- 
posed of a question or other requirement displayed to the student 
(and usually preceeded by a didactic statement of some kind), 
followed by a computer-waiting condition which allows the stu- 
dent to insert his response, which is in turn followed by a pro- 
gram reaction to that response. The program-reactive display 
may be subjoined by text which is really the beginning of the 
next frame, but the transition is not always apparent to the stu- 
dent : only the author/instructor needs to be aware of frame 
boundaries, in order to assemble his planned dialogue properly. 

For the purpose of this study, we envisaged a single frame 
with five « satellite » frames. Each of the satellites would be used 
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to supply an element of information requested by die student, 
and to ask him if the information sufficed. Tf so, he would be 
returned to the main frame; if not, he could elect either to pass 
to the next frame or to sign off. We call the whole complex of 
main frame and satellites a « macro-frame ». 

By calling up the satellites one by one, it is possible to provide 
independent subroutines which look like frames, i.e., each has a 
question, a wait mode, and a reaction, but which would never be 
used as instructional units by themselves. It would be possible 
to merge these mechanisms in the main frame, but this would 
require elaborate controls to prevent mutual interference. 

Another alternative is to maintain an instructional glossary in 
a separate file, to which the student has access at all times. This 
type of facility can be programmed in the more powerful systems, 
but its greater scope needs to be weighed against the advantages 
of defining a term in context. 

A flowchart of the A-Frame is given in Appendix B, and an 
abridged version of the actual machine program which would 
implement it in Appendix C. (Scoring devices and reiterative text 
are omitted throughout). 



Scansion 

Student input can be scanned for certain words or parts of 
words, in specified order or (if desired) in any order. Scan specifi- 
cations can also be nested (DISCUS) in such a way that input is 
tested in its entirety first for one set of elements, and, if successful, 
for additional sets of elements. This permits very eaborate 
decision trees. 

A « not » facility can also be used (DISCUS) to destroy any 
match condition when a specified unwanted element crops up in 
the response. This could lead to a solution of the problem of 
dealing with negatives, double negatives, triple negatives, etc., 
if an effective means of dealing with negative affixes can be 
developed. 

A series of scan specifications is always designed with a series 
of answers in mind. We have found it useful to categorize these 
along the following lines : 
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A L 



Ai 



Alp 



Alpm 



Alpf 



A l u 

F 



(Answer, Legal) 



(Answer, Illegal) 



(Answer, Legal, Perceived) 



(Answer Legal, Perceived, 
Matched) 

(Answer, Legal, Perceived, 
Failed) 



(Answer, Legal, Unperceived) 



(The sum of failed answers) 



A block, of student input 

(a) of limited length, 

(b) containing at least one 
term relevant to the 
question or its context, 
and 

(c) in syntactical order. 

Any block of student input 
not meeting the above re- 
quirement. 

Any A l which can be 
predicted by the author / 
instructor, as possible input. 

Any A lp which the author / 
instructor chooses to deal 
with uniquely. 

Any A lp which the author 
deems so remote or so in- 
consequential or so difficult 
to deal with in its existing 
form that it should be failed. 

Any A l which the author 
fails to predict as a possi- 
bility. 

Ai + A lu + Alpf 



The type of answer which principally occupies our attention 
is A lpm . It is not necessary that the author have in mind every- 
thing that the student might say embodying the A lPM element(s). 
It is necessary that he select scan elements which are likely to 
match with a broad range of responses relevant to the question. 
In other words, he formulates a key word or phrase which will 
match members of a set of semantically similar input, and which 
will distinguish semantically dissimilar input by failing to match 
with it. 

The range of A LPM is directly influenced by the form of the 
questions which precede them. A question posing a very rigid 
requirement (e.g., True-False) reduces the number of A lpm to 
a minimum, but ai the same time it vitiates the conversational 
mode. Our current goal is to open up questions and to give the 



7 



STUDENT FEEDBACK IN CAI 



227 



student as much freedom in formulating his response as we can, 
without letting match probability fall below acceptable levels. 

Appendix D lists the statements which would be displayed to 
students in response to various A LPM and F answers in the 
A-Frame, both initially (first pass) and recursively (subsequent 
passes). Other elements of the program’s reactions may be 
construed by referring to Appendix B and to the supporting 
material in Appendix C. As demonstrated by the imaginary 
dialogues at the end of Appendix D, it would be possible for the 
student to get through the frame very rapidly indeed. On the 
other hand, it would also be possible for him to flounder around 
in it for several minutes. 



The Statement/Question 

In a pamphlet entitled Microfiche Systems Planning Guide (1) 
(J. M. Baptie, ed.), appear the following statements : 

There are three basic types of unit microforms. These are : (1) the 
transparent film sheet, or microfiche, with a moderate to high page 
capacity; (2) the opaque Microcard R , with a moderate to high page 
capacity; (3) the aperture card, with a low page capacity... (p. 4) 



and 

To date, the significant microfiche applications appear to be : (1) distri- 
bution of technical report literature... (p. 3) 

These statements were reformulated into the statement /question 
on the student form (Appendix A), i.e.. 

There are three basic types of unit microforms. These are : 

(1) The transparent film sheet, or microfiche, 

(2) The opaque microcard, 

(3) The aperture card. 

Which of these do you think is the most suitable for dissemination of 
technical literature ? Why ? 

In the absence of extensive preparation, this is obviously a 
very faulty combination. What does the teacher mean by unit 
microforms ? Is microfiche the same as transparent film sheets or are 



(1) Copyright 1965 by Microcard Corporation, West Salem, Massachusetts. 
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they offered as two different things ? What are the microfiche, 
the opaque microcard, and the aperture card really like, anyway ? 
By « which » does the teacher mean « which one ? » What scope 
of dissemination is meant? Suitable for whom? What are the 
conditions that generally govern the dissemination of technical 
literature ? 

To cap it all, the instructor asks « Why ? », which would be an 
excellent question element if it weren't preceded by such a con- 
fusing array. 

It might be possible for a person armed only with an amateur's 
knowledge of photography and with the nature of technical 
literature to come up with the correct answer. However, it was 
not our purpose to establish this, but rather to see whether all 
respondents might be led to a degree of understanding which 
would permit them to formulate a correct response. This seems 
to accord with the theory that (at the problem-solving level) 
learning acquired through active participation by the student is 
more useful and durable than that which is passively ingested 
by him. 



Experience with the « A » Set of Answers 

As stipulated, twenty-five answers from the 50-answer sample 
were used to test the scans (SC) and their associated reactive texts 
in the « A-Frame ». It was expected that the « A » set of answers 
would reveal omissions and unforeseen ambiguities latent in the 
scan specifications. For example we might have specified 
(SC) CLARITY, LEGIBLE, CLEAR, 

EASY EYES, EASIER EYES 

with the idea of picking up something like 
It’s better because it*s more legible. 

But then the occurrence of 

« It’s better because it’s more readable » 

among the sample answers would show the need to add READ- 
ABLE to the specification. 

Also the answers were expected to turn up instances wherein 
the program reaction text should be modified to accommodate 
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variant input expressions. For example suppose we wanted to 
treat alike all answers expressing preference for microfiche 
because it is readily reproduceable. We might write the reactive 
text in the following form : 

« YES, IT IS ». 

Then if one of the samples turned out to be 

« I prefer microfiche because 9 simple copy machine can make copies very 
quickly and cheaply » 

clearly the reactive text would need to be adjusted. (Sometimes 
the reaction logic goes completely awry and new subroutines 
have to be added. In other cases the change of a single word will 
suffice to make the reactive text more broadly suitable). 

Both of these expectations were fulfilled, but we also discovered 
that the means for dealing with the « I don’t know » family were 
completely inadequate. And this led to one o c the most interesting 
developments to come out of the experiment. 



The « I don’t know » Condition 

The idea had been to scan for versions of « I don’t know » 
occurring with any of the following essential elements : 

DISSEMINATION (TECHNICAL LITERATURE) 

APERTURE (CARD) 

UNITIZED (MICROFORM) 

(micro) fiche (transparent sheet) 

(micro) opaque (card, microcard) 
and to deal with them in the five satellite frames. If more than 
one of the separate elements occurred in an « I don’t know » 
answer, an appropriate succession of satellites would be called up. 

Expressions of plain « I don’t know » or its variants would 
invoke a response asking the student to identify the terms he 
needed to have explained. 

Expressions of « I don’t know » together with unidentified 
matter would invoke a similar response, also letting him know 
that a portion of his answer had not been understood. 

Altogether, eight of the twenty-five answers fell into these last 
two categories ! 
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Group I — Don't know — no further clue 

#1 I don’t know. 

I don’t know. 

4^9 I really haven’t the slightest idea. 

Group II — Don't know — plus unidentified matter 

4^3 I don’t know enough to decide. 

# 4 l don’t know what these are exactly. So can’t tell. 

4^5 I don’t know. They all seem about equivalent to me. 

#10 Not enuf info to answer this — would need to know properties 
of microforms and kinds/quality of available photocopy. 

4 fcn IVe only had occasion to use the microfilm on reels, so I can’t 
really say. 

A ninth answer, « I can’t honestly compare because I am not 
familiar with all 3 of the media », really belongs in Group II, 
except that it was caught accidentally by the scan for « I don’t 
know » + « aperture card (3) ». 

On revision, we sought to identify the clues in #5, #10, 
and #ix and to modify the scans as follows : 

#5 — scan for « all, same, equivalent » and provide explanatory 
routine (probably a satellite). 

#10 — add « enuf » and « info » to the « don’t know » specification. 

1 1 — scan for « reels, rolls », etc. Although microfilm on reels is 
outside of the problem, it is relevant and may provide a good 
insertion point inside of the problem; so we would write an 
additional statement covering it. 

The original programmed response to Group I was « I’ll be 
glad to explain any of the terms I’ve used, if you wish. Just 
type ‘Clarify — ’ ». 

The original programmed response to Group II was « I don’t 
understand exactly what you need to have clarified ». 

Answer #10 would have fallen through to fail (another 
accident) because « enuf » and « info » had been omitted from 
the scan specification. However, as luck would have it, the 
programmed response was not too bad : « How’s that again ? 
I don’t understand your response as worded ». 

But all three responses fail to differentiate between cognitive 
states which would be fairly obvious to a human interlocutor. 
Granted that we can patch up #5, #8, #10, and #11, the 

II 
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i^maining five raise some interesting theoretical points which 
we are now obliged to deal with. 

Information-state, Motivation, and Motivational Indicators 

For the purpose of this study it has seemed reasonable to make 
certain assumptions regarding the information-state signalled 
by « I don’t know ». Without really plumbing the depths of 
cognitive theory, we might say that : 

1 . Information, in order to be recognizable as such by more than one person, 
must be capable of being expressed in terms which mean approximately 
the same thing to others. This set of agreed meanings is the basis for 
semantic transfer. 

2. Awareness of information requires a substrate, comprised of relatable 
units... (i) the presence, absence, and quality of which tend to bound the 
ultimate knowledge state. 

3. The cognitive process (either self-gene rated or derived from an outside 
source) consists of a perception of some kind of logical structure formed 
by the relationships between the information units. The integrity of this 
perception depends on the individual’s synthetic skill and on the 
accuracy/completeness of the substrate. Its transferability depends on 
semantic equivalence (1, above). 

4. Self-sustaining cognition can be triggered by an outside stimulus or by 
an influx of critically related information units. In either case it represents 
a learning process hich engages the student’s whole attention. Derived 
cognition, on the other hand, while it may be accepted, probably does 
not command the student’s attention to the same degree. 

The order in which the first three assumptions are stated is 
not meant to imply an unvarying sequence. For example, gaps 
in the informational substrate do not preclude formation of a 
knowledge structure, even though the structure may be somewhat 
tentative and unstable until the gaps are filled in. Nevertheless, the 
existence of some substrate appears to be prerequisite to structural 
perception, so a progression from (2) to (3) can be stated as a 
fifth assumption. (In fact, we might think of structural perceptions 



i? (1) Ausubel might speak of these as « anchoring ideas ». See D. P. Ausubel, 

| Educational Psychology; a Cognitive View (New York, Holt, Rinehart and 

I Winston, 1968) (esp. « Learning and the availability of relevant anchoring 



ideas », p. 132 et seq.). 
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being packed down in use until they can be employed as in- 
formational units, to serve as substrate elsewhere. But the reverse 
of such a process is hard to imagine). 

Probably none of the assumed conditions could be brought 
about in the absence of some kind of motivation, and it appears 
reasonable to suppose that this element is strongest at the moment 
of creation, i.e., when the structure is about to be perceived. 
Can we, then, use motivational indicators to tell us something 
about the student’s cognitive state ? For example, from expressions 
of disinterest that a semantic block exists, or that the informational 
substrate is so fragmentary that nothing can be made of it ? 

Does a request for specifics indicate that the structure - — though 
shaky • — is beginning to take shape ? 

These questions impel us to see if we can detect motivational 
indicators imbedded in « I don’t know »s, hoping that from these 
we might get an idea of information-state. This in turn would 
permit us to react to individual « I don’t know » answers with 
more discrimination. 

Reverting to the Group I answers : 

#i I don’t know. Curt. No evidence of interest. 

#z I don’t know. Possibly none of the terms in the 

statement/question connect with 
any information unit. 

#9 I really haven’t the slightest idea. Almost explicit disinterest, plus 

overtones of distaste for the 
system itself. There is a possibility 
that the student has some infor- 
mation but prefers to dismiss it, 
for the sake of being expressive. 
In any case, we have to take him 
at his word. 

In the face of zero motivation implying a near-zero informational 
substrate, we have two choices : 

We could say « All right, let’s talk about something else », 
and go to another part of the program — trying to find something 
which the student knows enough about to get off to a fair learning 
start. Later, he might become interested and knowledgeable 
enough in « microphotography », for example, to allow us to 
return him to the A-Frame on a better footing. (This faculty of 
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letting the student, influence his own learning path is of course 
one of the most promising features of CAI technology). 

We could start building, bit by bit, the necessary collection of 
information units by literally imposing them on the student — a 
difficult and thankless task, and one which promises no early 
surge of motivation on his part. 

In this particular study we attempted a modification of the 
second tactic in order to be in a position to observe second-pass 
performance later on. We tried to pinpoint critical information 
units which are necessary in dealing with the question, i.e., 
characteristics of microform types. Then we pointed out our 
awareness of the difficulty of answering the question without 
these units. Finally we presented a tactful offer of assistance. 

« I realize that it is almost impossible for you to answer without knowing 
something about the types of microforms mentioned, and about the factors 
involved in dissemination of technical literature. Fd like to help you tackle 
the problem from the beginning by supplying the information you need. 
Are you willing to work with me toward a correct solution ? » 

Second-pass reaction to a « yes » from the student would be : 
« Which item would you like me to describe first ? Type « De- 
scribe — ». Second pass reaction to a « no » from the student 
would be a signoff, with a consultation recommended. 

Turning to the answers in Group II : 

#3 I don’t know enough to decide. Polite regret implies some wil- 
lingness, so there may be a few 
#4 I don’t know what these are ex- scraps of information upon which 
actly — so can't tell. to build. 

Rather than trying to ascertain the areas of complete ignorance, 
the first move was to ascertain and strengthen the areas of 
knowledge, then to fill in the gaps. The following text was aimed 
at accomplishing this : 

« Well, suppose we start by examining the items you may already have heard 
about. I may be able to add some useful information about them. How 
about microfiche ? Microcard ? Aperture card ? The problem of dissemi- 
nation of technical literature ? Are you acquainted with any of these ? » 

Second-pass reaction to « Yes » would be : « Which ? » Second- 
pass reaction to « No » would be to step down to the text given 
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for plain « I don’t know » answers. Second-pass reaction to a 
specific would be to branch to a satellite. Third-pass reaction 
to another « I don’t know » would be something like this : 

« Shall we take them up one at a time ? Suppose we start with the problem 
of dissemination of technical literature ». 

This would be followed by at least four of the satellite sub- 
routines. 

To continue with Group II : 

#5 I don’t know. They all seem about Here the student is clearly making 
equivalent to me. an effort of some kind. He may 

already have a fair idea of the 
#8 I can’t honestly compare because dissemination problem, since his 
I am not familiar with all 3 of the response shows a narrowing to 
media. the three forms suggested. 

In this case we are justified in concentrating initially on a de- 
finition of the three types. 

A-Frame performance with other types of answer 

In the remainder of thir. section are listed the other answers, 
and an indication of how the program handled them. In order to 
save space at this point, we simply refer opposite each to the 
Appendix D code number which defines the reaction that would 
have occurred. An assessment as to whether or not that reaction 
would have been appropriate appears in the right-hand column. 



Group 

# 


Answer 

# 


Student input 


Appendix 

D 

channel 


Appro- 
priate 
1 st puss ? 


IV 


6 


I don’t know what an aperture 
card is. Nor (1) or (2). 


8 .a 


Yes 


IV 


8 (a) 


I can’t honestly compare because 
I am not familiar with all 3 of the 
media. 


7.d 


No 



(a) Repeated here for statistical purposes. (The program would have interpreted the input 
as meaning « I don't know what an aperture card is ». The outcome might have been 
satisfactory in this case, but no credit can be taken). 
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Group 

# 


Answer 

# 


Student input 


Appendix 

D 

channel 


Appro- 
priate 
1st pass ? 


IV 


12 


Please define opaque microcard 
and aperture card. 


8. a 


Yes 


V 


13 


I believe the opr.que microcard 
is the most useful. 


4.a 


Yes 


V 


14 


Transparent film sheet; just 
seems easy to store and use. 


2.a 


No (b) 


V 


15 


Aperture card too awkward for a 
big file, easier to enlarge micro- 
fiche for convenient viewing. 


4.b 


No (c) 


VI 


16 


None. I believe in hard copy. 
I dislike sitting in dark rooms 
and looking at idiot boxes. 


lO.a 


Yes 


V 


17 


I think the microfiche would be 
best because I think it can con- 
tain more info than the other 
forms. 


2.a 


No (d) 


IV 


18 


3. The aperture card can be 
handled mechanically. 


4.b 


No (e) 


IV 


19 


Aperture card, because the cita- 
tion infro (sic) is readable without 
resorting to a reader, 


4.b 


No (f) 


IV 


20 


3. Aperture cards because they 
are processed easily. 


4.b 


No(g) 



(b) Should have gone to 2,b.2. « Store » was overlooked in the scan specification. 

(c) Program fault. On revision, something to screen correct negative judgements is needed. 

(d) Should have gone to 2.b.2. « Contain more info » was overlooked in the scan specification. 

(e) « Why ? » will be misinterpreted. A scan for the sense of machine handling and processing 
is required. 

(f) « Why ? » will be misinterpreted. A scan for the sense of eye-legible tiding is needed 
in any case, and perhaps « citation » should be one of the specified keywords, although 
its occurrence seems remote. 

(g) Same as (e). 
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Group 

# 


Answer 

# 


Student input 


Appendix 

D 

channel 


Appro- 
priate 
1st pass ? 


IV 


21 


The opaque microcard is ade- 
quate. 


4.a 


Yes 


IV 


22 


The microfiche offers better re- 
production (visually) than the 
microcard. 


2. a 


No (h) 


IV 


23 


I would suggest the microfiche, 
since complete information can 
be stored in a small space. 


2.b.2 


Yes 


IV 


24 


#2 is most suitable. As a 
positive copy of the text it is 
least objectionable to the user. 


2. a 


No (i) 


IV 


25 


The microfiche has proved quite 
useful for cartographic informa- 
tion. 


2.a 


Yes (j) 



(h) The answer itself is ambiguous, but suggests the need for scans covering (1) manual 
copying, (2) rendering into hard copy, and (3) clarity of display in a reading machine. 

(i) A scan is needed to dispose of the positive-negative issue, if possible, followed by a 
return to other considerations. 

(j) « Cartographic » is too far out to provide for at present. One can only hope that the 
« Why ? » will elicit something which the system can process more meaningfully. 



Evaluation and Revision of the A-Frame 

Appendix F shows in tabular form the characteristics of each 
of the « A » answers; how it fared in the « A-Frame »; and an 
evaluation of the A-Frame as to each, on a scale of i-o. The 
summary of this data leads to one conclusion, namely, that although 
the percentage of matches on the first pass was high (y 6 %), frame 
performance as a whole was not commensurately good , since program 
reactions were suitable in only about half of the cases . Accordingly we 
were obliged to evaluate the frame as a whole at less than 50 %. 

Let us examine this more closely. The A-Frame was planned and 
written by an experienced CAI programmer, and his work was 
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checked and emended by a second person, also experienced in 
the field. It is not surprising that between them they should have 
foreseen a range of keywords such that 24 out of the 25 answers 
were intercepted. But they failed to anticipate many of the sec- 
ondary elements which were needed to refine the meanings of the 
answers. For example, every response containing a primary 
element : microfiche , microcard , or aperture card y was identified as 
such, but further processing did not occur, because of various 
omissions and other faults in dealing with such things as « me- 
chanical » and « info ». 

This in itself was not unexpected. The original purpose of the 
test was to see how much of an improvement could be made 
in the frame, using ideas suggested by operational experience. 
Admitting that no one person nor even two persons in collabora- 
tion could imagine the whole scope and variety of possible re- 
sponses to a fairly open question, we wanted to gauge the direct 
utility of student responses in augmenting the range of our 
perceptions in this regard. We might even decide that the best 
way to design a frame was to erect only the barest structure, 
without trying to cover every eventuality, and to rely on an 
influx of on-line responses from live students to point the way 
to the frame’s further development. 

But the experience with « I don’t know » answers raised 
another problem. It was not that the program reactions were 
particularly bad. They just weren’t particularly good. They made 
no account of attitudinal overtones which to a human teacher 
would have clearly indicated shades of knowledge-condition in 
the student. The effort to improve this situation needs to be 
considered apart from the simple refinements mentioned in the 
preceding paragraph, because, as indicated in the discussion of 
cognitive states, we believe that such differentiations might prove 
generally useful as tools in CAI methodology. 

In order to investigate this, all the words and phrases which 
might be expected to have key status in a « I don’t know » 
statement were listed. Items were clustered in various ways, and 
the clusters (of which there were finally ten) were mapped into 
each other on a trial and error basis until it was possible to discern 
roughly three levels of attitude. These combinations were then 
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assigned programmatically, yielding an array of scans we call the 
« M-series », set forth on page 276, A ppendix G. 

Taking the scan labelled « M 0 » (motivation zero) as an ex- 
ample, we see that it is composed of elements from the scan 
labelled «-Ai », or from scans « A2 » plus « Bi » plus « Ci ». 
Any answer which had not been screened out by an earlier scan 
would thus be matched by « M 0 » if it contained any of the 
following : 

... don’t know... 

... can’t say... 

... can’t imagine... 

... you expect... 

... you think... 

... at sea... 

... with it... 

... search me... 



or a combination of the following : 

... don’t have... 
... not have... 

... haven’t... 

... expect have... 
... think have... 



plus 



plus 



... least... 

... vague... 

... vaguest... 
... slightest... 
... fuzzy... 

... fuzziest... 
... foggy... 

... foggiest... 
... remote... 
... remotest... 



. idea... 

. notion... 

. interest... 

. concept... 

. conception 
. inkling... 
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Thus 

« What makes you think I could possibly answer that ? » 

« Search me 1 » 

« I’m sure I don’t know ». 

« I haven’t the foggiest notion ». 

would all be matched in « M 0 » and be treated accordingly. 

The « M-Series » of scans would not need to bulk as large in 
the program as might be supposed from the foregoing, because 
of the common device of forgiving parts of words which may 
be missing or misspelled. For example the form fogg* (pilot) or 
* fogg’ (discus) accepts « foggy », « foggiest », « foggier », 

and « fogged ». A third language, foil (i), forgives a specified 
percentage of characters : foggiest (50 %). 

The adoption of an additional line of inquiry did not seem to 
prejudice the original idea of the model, because it is possible to 
assess performance according to A L groups individually or in 
combination. Thus in the A-Frame the following breakdown can 
be made : 





Score 


Possible 


% 


Groups 1 and II (« I don’t know ») 


5 1/2 


8 


69 


Groups III, IV, & V (all others) 


6 1/2 


17 


38 


Composite 


12 


25 


48 



The A-Frame’s superficial success with Group I and II rests 
solely on the fact that forms of « I don’t know » can be ex- 
haustively predicted, compared with all the other things that can 
be said by a student, and a response that makes some sense can 



(1) Hesselbart, J. C., « FOIL — a file-oriented interpretive language » in 
Proceedings of the 23rd National Conference of the Association for Computing Machinery 
(ACM Publication P-68, Princeton, Brandon, 1968), pp. 93-98. See also in the 
same publication the review of CAI languages by Karl Zinn, « Programming 
conversational use of computers for instruction » (pp. 85-92). 
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be trotted out. The M-series of scans represents a new objective, 
the hope that we could significantly refine this ongoing sense. 

As for Groups III, IV, and V, we could continue to pursue 
the general intent of improving scan specifications word by word, 
so as to catch and channel a greater variety of unique responses. 
In some cases it would be necessary to write separate new scan 
statements and program reactions to deal with aspects we had 
completely missed in the first version, such as the idea of me- 
chanical handling. 



Part II 



Procedure 

Appendix G (Flow Chart) and Appendix H (Program) show 
the model as revised and redesignated the « B-Frame ». 

Very little was done to alter the flow, except to insert the 
M-series of scans just ahead of « dunno 4* garb », and to add 
a satellite to deal with mechanical handling. The A-Frame flow- 
chart omitted certain controls specified in the program, and the 
B-Frame does the same. 

Appendix I corresponds with Appendix D, in specifying 
exactly what would happen to an identifiable type of input marched 
by a particular scan statement. 

All this work was completed before we allowed ourselves to 
look at the « B » set of answers. When we had everything ready, 
we proceeded to drop them one by one into our paper « hopper ». 
It may be asked why we didn’t actually put the program in the 
computer and test it on-line, in order to avoid any possibility of 
shading the results. We would have liked to do this, of course, 
but as mentioned in the beginning we had assumed an augmenta- 
tion of an existing language (PILOT) with features of one which 
was not yet operational (DISCUS). In other words, we were 
using a non-existent CAI language. More recently, DISCUS has 
indeed been implemented at the University of California Berkeley 
and UCLA campuses, and we expect to use it for most of our 
CAI programming. 

Anyone monitoring the project might have objected to the 
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fact that we did not seem to require an absolute fit of some input 
terms with the scan specifications, as originally laid down in 
Appendix C and Appendix H. The reason for this is that we 
assumed the « forgive » convention whereby « stor* », for 
example, would match « storing », « storage », or « stored», but 
wrote one of these out for better understanding by the reader. 
This is obviously misleading, though, so we revised these two 
Appendices to show such a specification as « jjforage » with only 
the underlined characters representing the absolute requirement. 
Thus a reader can perceive both the real specification and the 
term idea behind it. Except for this reformatting, no retrospective 
changes of any consequence were made in any of the supporting 
documents. 



Comparison of Results 

The summaries of A-Frame and B-Frame observations are 
compared in Table 1. Clearly a remarkable improvement took 
place in respect to what we call « sense score », i.e., the percentage 
of student answers to which the program would have responded 
satisfactorily. 

A satisfactory response in every case is deemed to be one which 
not only makes sense on the first pass but offers a good chance 
of leading to a sensible response on the next pass. (This rules 
out the plausible but misleading response of « Why ? » to ques- 
tions A-18 and A- 1 9 (see page 235) where the student is talking 
about one tiling and the program is talking about another). 

In these assessments the student viewpoint controls; we cannot 
excuse awkward — « noncommunicating » — responses on 
grounds that they suit our pedagogical plan or that they will lead 
in the end to some kind of complete exposition. 

Before looking into the significance of the results, we need to 
ask if enough similarity existed between the A set of answers 
and the B set of answers to validate a comparison of A-Frame 
performance and B-Frame performance. I think we can say that 
the motivation scores and the proportions of Group V answers 
show that the two sets were quite alike — certainly close enough 
for our purposes. 
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Comparison of A-Frame and B-Frame Performance 
on Sim/lar Sets of Answers 





A-Frame 


B-Frame 


Change 


Match score 


24 (96 %) 


25 (100 %) 


+ 4 % 


Sense score 








Groups I & II 


5 1/2/ 8 (69 %) 


5 1/2/ 6 (91 %) 


+22 % 


Groups III, IV, V, & VI 


6 1/2 / 17 (38 %) 


16 1/2 / 19 (86 %) 


+48 % 


Composite 


12/25 (48 %) 


22/25 (88 %) 


+40 % 


Characteristics of the sets : 


Motivation 0 


4 x 0=0 


2 X 0 = 0 




Motivation 1 


4x1=4 


6x1=6 


+2 


Motivation 2 


3x2=6 


3X2=6 


— 


Motivation 3 


14 x 3 = 42 


14 x 3 = 42 


— 


Composite 


52 


54 


+2 


Possible maximum 


75 


75 


— 


Factored 


52/75 


54/75 


+2/75 


Answer Type Groupings : 


I 


3 


1 


— 2 


II 


5 


5 


— 


III 


1 


0 


—1 


IV 


3 


6 


+3 


V 


12 


13 


+ 1 


VI 


1 


0 


—1 




25 


25 


— 



Significance 

The total number of unique A L which might be entered in 
response to a given question is a function of the openness of 
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that question, and can range from a very small set (as in response 
to a True-False) to one which is very large (as in response to 
« What’s it all about ? »). Let us call this number K and charac- 
terize it as finite (because of the constraint of our A L definition) 
but indeterminate (because no matter how many different legal 
answers we can imagine in response to a question, someone may 
come along with one we had not thought of). 

If we were to scan randomly for A L the curve of match pro- 
bability would be a straight line as shown in Figure 1. But we 
do not scan randomly, because we know that the frequency of 
occurrence of some A L is far greater than that of others, and that 
it is possible to attain a better match probability for K scans by 
tailoring them to the more frequent A L . The uppermost curve 
(line 4) represents the cumulative frequency of occurrence of 
A l and at the same time serves to represent the match proba- 
bility (P) which a CAI programmer could obtain for N scans if 
he knew the makeup of the items to the left of N. 

But of course he doesn’t know this. He can only guess, on the 
basis of his perception of (1) the topic, (2) the student set, (3) the 
position of the question in context, and (4) the language in use, 
which A l will occur more frequendy than others, and how 
much more frequendy. The richer these variables, the harder 
it will be for him to predict not only the identity of the A L to 
the left of N but the sh?ne of the curve of their frequency. Line 2 
shows how his prediction will always be better than random 
but less than perfect. 

Line 3 represents a closer approach to Line 4, attained by virtue 
of revising and improving upon the original, highly subjective, 
work of the programmer. What is the most efficient way of doing 
this ? By bolstering his predictions with external data furnished 
him by the system in the course of its operations. 

We would like to get a better idea of the role of this external 
data (feedback) — what does it look like ? how can it best be 
categorized? how many iterations should be considered before 
the utility of the process is overshadowed by the tasks associated 
with it ? These questions have a direct bearing on the refinement 
of conversational CAI techniques. 

2 * 
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Fig. I 

We are still a long way from being able to scale curves 2, 5, 
or 4, for even one set of variables, but at least the results obtained 
from this study indicate that 

(1) for an open question with a large K and an adult, heterogenous student 
set, a striking improvement in meaningful intercept can be obtained 
through close analysis of a modest number of live responses, and 

(2) as a corollary, the rate of improvement to be derived from close analysis 
beyond this modest number must decline quite rapidly. 

It is only fair to ask how much of the improvement in meaning- 
ful intercept (which is a better measure than raw match proba- 
bility) should be credited to the M-series of scans, representing 
a completely new conceptual element in the B-Frame. Looking 
back to the evaluations given for responses to Group I and 
Group II answers in the A-set, it will be seen that none of them 
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was downgraded for failure to detect nuances of « I don't know », 
but only for mechanistic faults. Thus the conditions determining 
sense score are the same for both frames. 
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APPENDIX A 



Forms 

1. Attachment (i) is the form of answer sheet used by all fity 
« students ». 

2. Attachment (2) is the form used by the authors for analysis 
of student responses and program reactions. 

Attachment (1) 

From : J. C. Meredith, Institute of Library Research 
(Information Processing Laboratory Project) 

You are asked to participate in a survey of response patterns to a single 
statement/question typical of programmed instruction, as follows : 

There are three basic types of unit microform. 

These are : 

1. The transparent film sheet, or microfiche. 

2. The opaque microcar d< R >. 

3. The aperture card. 

Which of these do you think is most suitable for 
dissemination of technical literature ? Why ? 

(Assume that the program is capable of dealing with responses consisting of 
symbols, numerals, or English words, singly or in combination, up to two lines 
in length. Your response may be in the form of a statement or a question or 
both, but should at least be relevant. Try to decide on your answer quickly.) 

Response : 
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Thank you. Please check the appropriate box(es) : 



□ LS 

□ ILR 

□ Other 



Attachment (2) 

STUDENT ANSWER 
PROGRAM RESPONSE 

1. First pass 

2. Likely 2nd pass 

SATISFACTORY ? 

TYPE OF MODIFICATION INDICATED 
REVISED PROGRAM ELEMENT 



« A-Frame » Flowchart 

1. Attached. 

2. The chart represents in abbreviated form the decision sequences 
envisaged for the « A-Frame ». The following notes pertain : 

1) The « Q » entry (large circle) represents student input, terminating 
with a « send » signal which causes the computer to resume 
processing. 

2) « Q » returns (small circles) imply a textual display of some kind, 
generated according to the specifications laid down in the program 
itself, and held on the CRT while the computer is waiting for the 
next input. Consult Appendix D to ascertain actual text planned for 
use in various situations. 

3) « OUT » (small circles) signifies exit to the next frame. 

4) To avoid excessive complication c£ the flowchart, several tests 
(diamonds) have been omitted. For example, the first five major 
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tests to the left of the « DUNNO » diamond entail additional 
subordinate tests and decisions associated with recursive scan of the 
same input string, under a series of « nested » scan parameters. 
These are omitted from the diagram. 

5) The group of operations to the left of « DUNNO » is devoted to 
provision of information intended to help the student solve the 
problem. One of five information « satellites » — UNITZ — is 
shown in the large box. It should be visualized as occupying the 
action box marked « CALL UNITZ » (hand). 

6) The group of operations to the right of « DUNNO » is designed 
to accommodate a student’s attempt to make the right choice. Each 
choice and the first recognized valid reason, or a combination 
(not shown) of the two or three best reasons, call forth a unique 
block of text. Note that if the student fails to give a recognized 
reason, the program loops through to a display of « Why ? » to Q, 
and his second response is scanned without requiring him to repeat 
his choice. (This is another case where the diagram omits specific 
tests). 

7) The remaining operations (bottom center) are miscellaneous tests and 
utility operations, ending with two fail-match displays terminating 
the frame. 

8) The typed numerals refer to answers in the « A » set, located so as 
to show how they would have been treated in a first pass through 
the program. 
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The « A-Frame » Program 

1. Attached. 

2. The program listing resembles actual source input in a high- 
level CAI programming language (selected string match type), 
such as PILOT. We have, however, omitted all delimiters (which 
serve only to notify the compiler whether it is dealing with a 
label, an operation code, or an operand) and have stylized the 
scan specification in logical form (e.g., a comma means « or »). 

3. The following notes will assist the reader in interpreting the 
program : 

1) All operations are sequential from left to right and from top to 
bottom. 
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2) For purposes of this sample, « G » signifies a display occurring 
only if a preceding scan specification has been successfully matched. 
« F » signifies a display occurring only if all match attempts have 
failed; and « T » signifies a display occurring regardless of match 
condition if execution reaches that point. 

3) Reference to a label in quotes means that everything in the operand 
so labelled is to be inserted at that point. The expanded versions of 
all labelled alphabetic strings are listed toward the end of the program. 

4) The control statements in the right hand column would in many 
cases be constrained to operate only under a « match » condition, 
the same as a « G » textual display. The necessary qualifiers for this 
have been omitted for the sake of simplicity. 

5) Control statements ADD, TALLY, and MARK mean «add», 
« add one », and « ensure one and one only in » respectively* They 
affect the contents of numerical counters whose existence is estab- 
lished implicitly rather than by formal definition. 

6) The sequence in which various match attempts are made reflects the 
rule that normally scans terminate immediately on successful match 
— the remainder of student input is ignored. This favors putting 
the larger, more difficult specifications ahead of those which are 
easier to satisfy. For example, the scan for « 1 » is placed at the 
very end, so that it will not ensnare emphatic statements which 
contain sought-for elements specified in other scans. 

7) Many of the planned unique text displays ate so similar in structure 
that they can be assumed. For example the response to «I like 
microfiche because it’s convenient » would call forth exactly the 
same kind of text as « I like microfiche because it’s cheap ». To avoid 
needless repetition, many of these have been omitted from the listing. 

8) A sample satellite is shown on the last two sheets of the program. 
The other four satellites would resemble it very closely. It might 
even be feasible to use just one satellite with a choice of texts to be 
« USED » depending on which indicators (variables such as 
MICRO Y) had been activated. 
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Reaction Text Displays — « A-Frame » 

1. List attached. 

2. This Appendix lists the textual responses programmed in the 
« A-Frame », that is, the responses which are called forth from 
the computer as part of its reaction to student input. Roman 
numerals indicate « pass status » as follows : 

I. Responses which can occur on the first pass (i.e., in answer to a students 
first input), or on subsequent passes. 

II. Responses which can occur only on the second or subsequent passes 
(depending on certain conditions being set in an earlier pass). 

III. Responses occurring on the second or subsequent pass if a satellite has 
been called. 

All student answers are classified A LPM (legal, perceived, matched) 
except numbers 12, 27, 28, and 31, which are F (failed), being 
either A r (illegal), A LU (legal but unperceived), or A LPF (legal, 
perceived, but intentionally failed). 

3. Types of student answer are indicative only. For example, each 
type or subtype listed represents hundreds or even thousands of 
different inputs all containing a required key element. For the 
exact specification of these elements, consult Appendix C. The 
designations « Right » and « Wrong » indicate the instructor’s 
viewpoint and have no mechanistic significance in the program. 

4. For the sake of brevity, many unique responses are left to the 
reader’s imagination, in situations where they would require 
only slight rewording of a response which has been written out. 

5. None of the text should be interpreted substantively. We 
expect an expert in the field might think of many counter-argu- 
ments. 

We are concerned here with dialogue rather than with factual 
distinctions, and in regard to the latter defer entirely to copy- 
righted reference 5. 

6. At the end of this Appendix are included two sample dialogues 
which might occur in an on-line situation, to illustrate the kind 
of interchange which might take place in the « A-Frame ». 
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7 . First pass . 

Type of answer (numbers and letters Program display reaction 
establish categories referred to in (« E » = « exit to next frame ») 
Appendix F) 

i. Right — both elements 

a. Reason : ease of distribution, You are correct in picking 

of mailing, of sending microfiche, and ease of distri- 

bution and use is one of its 
compelling advantages. 

(use FICHUP text) 

FICHUP : 

Microfiche can be duplicated 
very readily and inexpensively, 
and in fact is the standard in use 
by the federal government for 
distribution of Clearinghouse, 
DOD, NASA, AEC, and ERIC 
documents — which include a 
large portion of our technical 
literature output. Recipients 
can readily use them in a wide 
variety of readers, or (unlike 
opaque cards) render them back 
into eye-legible form through 
reader-printers. With aperture 
cards, several units are required 
to convey a multi-page docu- 
ment, hence they are more 
suitable for single items, such 
as a file of engineering draw- 
ings. (Aperture cards are more 
often used in a mechanized 
system, but this in itself does 
not make them more suitable 
than fiche, or less suitable). (E) 

b. Reason : cost, cheap, inexpens- You are correct in picking 

ive, save money microfiche, and cost consider- 

ations certainly enter into the 
picture. Microfiche can be du- 
plicated very readily and in- 
expensively (etc. — see 
FICHUP in r. a abeve). (E) 
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c. Reason : i.a + i.b Good. You correctly picked 

two of the important advan- 
tages of fiche. (FICHUP) (E) 



2. Right — first element only 

a. No reason given 

b. Wrong reason given 

(i) experience (« because I’ve 
had more experience with 
it »). 



(2) capacity (« because it has 
greater capacity »). 



(3) clarity 

(4) knowledge («the only one 
I know about »). 

3. Wrong — Two first elements 



Why? 

Can you compare them more 
on the basis of information ? 
If you would like to know 
more about the other forms, or 
about the general requirements 
for dissemination of technical 
literature, just type 

« Clarify ». 

You are correct in picking 
microfiche, but the reason it is 
superior for dissemination of 
technical literature is not that it 
has greater capacity than other 
unit forms (which depends on 
the reduction ratio in use) but 
on the fact that microfiche can 
be duplicated very readily and 
inexpensively. (FICHUP) (E) 
(similar to 2.b.a above) (E) 
(similar to 2.b.2 above) (E) 

See if you can pick just one, on 
the basis of what you can learn 
from me on the subject. 



4. Wrong — wrong first element 

a. Microcard Why ? 

b. Aperture card Why? 

5. Wrong — Don’t know 

a. Alone I’ll be glad to explain any of the 

terms I’ve used, if you wish. 
Just type « Clarify— ». 

6. Wrong — Don’t know + uniden- 
tified element 
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a. « Don’t know because 
(-?-)»• 

7. Wrong — Don’t know + single 
identified 

a. Unit, unitized 



b. Transparent, film, sheet, 
fiche, microfiche 

c. Opaque, microcard 

d. Aperture 

e. Dissemination, technical, 
literature 

8. Wrong — Don’t know + multiple 
identified 

a. Any combination of 7.a, 7.b, 
7 *c, 7-d, and 7.e) 



9. Wrong — Don’t know + general 
a. Any 



I don’t understand exactly what 
you need to have clarified. 



(First pass only) 

I understand you need more 
information about « unit » or 
« unitized » microforms. A unit 
microform is one which... 
(etc.) Does this resolve your 
uncertainty concerning the 
definition of unitized micro- 
forms ? 

(Second pass et seq.) I believe 
you already mentioned your 
problem with « unit » or « u- 
nitized » microforms, and I 
tried to explain. If you are still 
confused, perhaps you should 
sign off and seek help. 

(Similar to 7.a) 

(Similar to 7.a) 

(Similar to 7.a) 

(Similar to 7.a) 



I understand you need more 
information about (7.a, 7.b, 
7.C, 7«d, 7.e). We will take them 
up in that order. (Stacked sub- 
routines of 7.a-e) 



One of them is superior in re- 
produceability, cost, general 
use, etc. Do you need more 
information ? If so, type « Clar- 
ify ». 



A ^ 
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b. No experience 


Fd like you to choose on the 
basis of available information, 

if possible. Type « Clarify » 

for explanation of individual 
elements. 


10. Wrong — Rejection 




a. Dislike 


Aside from your personal feel- 
ings in the matter, can you 
express an opinion as to which 
might be least //^suitable for 
the stated purpose ? 


b. Expletive 


Really ? Fm surprised you feel 
so strongly about it. If you’d 
like clarification of any element 
of the statement about unit 
microforms, or of the question, 
just ask. Otherwise please sign 
off and notify the lab assistant. 


11. Wrong — Request 




a. Clarify, define 


(Same as 7 and 8) 


12. Fail 


How’s that again ? I don’t 
understand your response as 
worded. Please elucidate. 


Second pass 




1 3 . Right — both elements 


Not applicable (frame com- 
pleted on first pass) 


14. Right Second Element 
(first element right) 




a. Reason : ease of distribution, of 
mailing of sending 


(Same as i.a) (E) 


b. Reason : cost, cheap, inex- 
pensive, save money 


(Same as i.b) (E) 


c. Reason : i.a + i.b 


(Same as i.c) (E) 


15. Right Second Element 
(first element wrong) 




a. Microcard 1st element 
(1) experience 


(Similar to 2.b.i, 2.b.2, 2.b.3, 
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(2) capacity 
(5) clarity 
(4) knowledge 


and 2.b,4,with special wording) 

(E) 


b. Aperture card 1st element 


(Same as 15. a) (E) 


16. Wrong Second Element 
(1st element right) 

17. Wrong — Don’t know. 

18. Wrong — Don’t know + 


(ditto) 


(Repeat 5) 
un- (Repeat 6) 


identified 


19. Wrong — Don’t know -j- 




a. Single identified 


(Similar to 7) 


b. Multiple identified 


(Similar to 8) 


20. Wrong — Don’t know + general (Repeats 9) 


21. Wrong — Rejection 




a. Dislike 


(Repeats 10. a) 


b. Expletive 


(Repeats io.b) 


22. Wrong — Request 


(Same as 7, 8, and 11) 


SUBROUTINE FOLLOWING 7 




23. Right 

a. « YES » 

24. Wrong 


Fine. If you are in doubt about 
any of the other terms, don’t 
hesitate to ask. Here is the original 
question again, (repeats question) 


a. « NO » 


Rather than spend more CAI time 
on it now, I suggest that you 
either sign off and seek out an 




instructor, or type « Pass » and I 
will go to the next question. (E) 


SUBROUTINE FOLLOWING 8 


25. Right 

a. « YES » 

26. Wrong 


In that case we will pass to the 
meaning of ... (next occurring 
item of requested information). 


a. « NO » 


(Same as 24). 
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27. FAIL — If #12 unused on 1st (Same as 12), 
pass 

28. FAIL — If #12 used on 1st pass (Use 12 (admits frame failure; 

may quote all instructive text)). 



III. Third pass 

(Predictable only if a subroutine was used on the 1st or 2nd pass) 



SUBROUTINE FOLLOWING 8 and 9 



29. Wrong — Continue 




a. « PASS » 


All right. But please get clarifica- 
tion at the first opportunity. (E) 


30. Wrong — EXIT 




a. « I quit » 


You are now signed off. (E) 



31. FAIL — =27 & ^28 used (Program falls through to next 

frame) (E) 



A SHORT DIALOGUE AS ENVISAGED IN THE « A » VERSION 



Instructor 

Student 

Instructor 



Student 

Instructor 



Student 

Instructor 



(Statement and question) 

I don’t know what a microcard is, really. 
(Defines microcard) Does this resolve your 
uncertainty concerning the definition of 
microcards ? 

Yes. 

Fine. If you are in doubt about any of the 
other terms don’t hesitate to ask me. (Repeats 
statement and question). 

Microfiche, because it is cheaper to reproduce 
and distribute in quantity. 

You are correct in picking microfiche, and 
ease of distribution and use is one of its 
compelling advantages. Microfiche can be 
duplicated very rapidly and inexpensively ... 
(etc.) 



4-U 



(Exit to next frame) 
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A LONGER DIALOGUE AS ENVISAGED IN THE « A » VERSION 



Instructor 

Student 

Instructor 



Student 

Instructor 



Student 

Instructor 



Student 

Instructor 



Student 

Instructor 



Student 

Instructor 

Student 

Instructor 

Student 

Instructor 

Student 

Instructor 



(Statement and question) 

I dislike all three. 

Aside from your personal feelings in the 
matter, can you express an opinion as to which 
might be the least ///^suitable for the stated 
purpose ? 

How can I ? I don’t know much about any 
of them. 

One of them is superior in reproduceability, 
general use, cost, etc. Do you need more in- 
formation ? If so, type « Clarify ». 

O.K. Clarify what is a microcard. Also 3. 

I understand you need more information about 
opaque microcard — aperture cards. We will 
take them up in that order. (Defines micro- 
card) Does this resolve your uncertainty 
concerning microcards ? 

Affirmative 

In that case we will pass to the meaning of 
aperture card : (Defines aperture card). Does 
this resolve your uncertainty concerning aper- 
ture cards ? 

Yes 

Fine. If you are in doubt about any of the 
other terms, don’t hesitate to ask. Here is the 
original question again. (Repeats statement 
and question). 

The whole question seems unreasonable to me. 
How’s that ? I don’t understand your response 
as worded. Please elucidate. 

Well, I’ll make a guess. Microcards and aper- 
ture cards. 

See if you can pick just one, on the basis of 
what you can learn from me on the subject. 
Microcards. 

Why? 

Because they’re cheaper, I guess. 

No, as a matter of fact, they are more expensive 
than either of the others. Microfiche is better 
for dissemination of technical literature than 
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either microcard or aperture cards, because 
... (etc.) ... Microfiche can be duplicated 
... (etc. — FICHU P). 

(Exit to next frame). 



APPENDIX E 

« A » STUDENT ANSWERS AND PROGRAM RESPONSES 

This Appendix lists the student answers of the « A » set, 
together With the reactive text displays which would have been 
called forth in the « A-Frame ». All items are grouped according 
to « sense » from the instructor’s viewpoint. The several evident 
inconsistencies are discussed in the body of the paper. 



STUDENT INPUT 


REACTION TEXT DISPLAY 


Group I : No clue 


# 1 


I don’t know. 


I’ll be glad to explain any of the terms 
I’ve used, if you wish. Just type 
« Clarify ». 


# 2 


I don’t know. 


(same as 4M) 


# 9 


I really haven’t the slightest idea. 


(same as 4M) 


Group II : Polite regret some narrowing of information need 


# 3 


I don’t know enough to decide. 


I don’t understand exactly what you 
need to have clarified. 


# 4 


I don’t know what these are exactly, 
so can’t tell. 


(same as # 3 ) 


# 7 


Don’t have enough experience to 
judge. 


I’d like you to choose on the basis of 
available information, if possible. 
Type « Clarify » for explana- 

don of individual elements. 



A? 
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STUDENT INPUT 


REACTION TEXT DISPLAY 


#10 


Not enuf info to answer this — would 
need to know properties of micro- 
forms and kinds/quality of avail 
photocopy. 


How’s that again ? I don’t understand 
your response as worded. Please eluci- 
date. 


#11 


I’ve only had occasion to use the 
microfilm on reels, so I can’t really say. 


(same as # 1) 


Group III : « I don’t know » + attempt 


# 5 


I don’t know. They all seem about 
equivalent to me. 


(same as #3) 


Group IV : Specific information need 


# 6 


I don’t know what an aperture card is. 
Nor (1) or (2) 


I understand you need more informa- 
tion about aperture card - microfiche - 
opaque microcard. We will take them 
up in that order. Aperture cards are 
not true unit microforms since they are 
« assembled », generally from card 
stock, adhes : ”e, and film. They ate 
well suited to applications such as 
engineering drawings... (etc.) Does 
this explain the term adequately for 
you ? 


# 8 


I can’t honestly compare because I am 
not familiar with all 3 of the media. 


I understand you need more informa- 
tion about aperture cards. Aperture 
cards are not true unit microforms 
since they are « assembled '),... (etc.) 
Does this explain the term adequately 
for you ? 


#12 


Please define opaque microcard and 
aperture card. 


I understand you need more informa- 
tion about opaque microcard - aper- 
ture card. We will take them up in that 
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STUDENT INPUT 


REACTION TEXT DISPLAY 






order. Microcard is a registered trade- 
mark for the form which is also called 
the « microopaque », and is charac- 
terized by ... (etc.) Does this explain 
the term adequately for you ? 


Group V : Solution offered 


#13 


I believe the opaque microcard is most 
useful. 


Why? 


#14 


Transparent film sheet; just seems easy 
to store and use. 


Why? 


#15 


Apf 1 ure cards too awkward for a big 
file, easier to enlarge microfiche for 
convenient viewing. 


Why? 


#17 


I think the microfiche would be best 
because I think it can contain more 
info than the other forms. 


Why? 


#18 


3. The aperture card can be handled 
mechanically. 


Why? 


#19 


Aperture card, because the citation 
infro (sic) is readable without resorting 
to a reader. 


Why? 


#20 


3. Aperture cards because they are 
processed by machine easily. 


Why? 


#21 


The opaque microcard is adequate. 


Why? 


#22 


The microfiche offers better reproduc- 
tion (visually) than the microcard. 


Why? 


#23 


I would suggest the microfiche, since 
complete information can be stored in 
a small space. 


Why? 
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STUDENT INPUT 


REACTION TEXT DISPLAY 


#24 


#2 is most suitable. As a positive 
copy of the text it is least objectionable 
to the user. 


Why? 


Group VI : Other 


#16 


None. I believe in hard copy. I dislike 
sitting in dark rooms and looking at 
idiot boxes. 


Aside from your personal feelings in 
the matter, can you express an opinion 
as to which form might be least 
////suitable for the stated purpose ? 



APPENDIX F 



« A-Frame » performance summary 

1. Attached, 



2. Explanation of column headings follows : 

1) Student inswer number generally in random order, except that 

when typing the list from the answer 
forms, « don’t know »s were entered first. 

2) Student |inswer except for line breaks, a literal version 

V ip -each case. 



3) Sense group 



4) Scan fate 



5) Answer type 



ordered, I through VI (see Appendix E 
for designations). 

an identification of the particular scan 
specification which is believed would have 
achieved match condition, for the student 
answer given. (Note : seeming incon- 
sistencies can be resolved by considering 
(a) sequence of scans, together with (b) 
sequence of elements in the answers). 

classification of answers according to 
categories discussed on page 6. 



i 



5 1 



STUDENT FEEDBACK IN CAI 



271 



6) First pass program reaction refers to item numbers in Appendix D. 

7) Sense score an evaluation of performance (0-1) based 

on the following criteria : Does the first 
pass reaction make sense, and is the pros- 
pect of sensible second-pass reaction 
good ? 



8) Motivation rating assigned this is a rating intuitively assigned by the 
by the instructor instructor, based on the content and 

overtones of the student’s answer : 

M0 — no apparent motivation 
Mi = polite regret, some motivation 
M2 = fair-to-good motivation 
M3 = weighting factor for Group V 
answers, in which good to excel- 
lent motivation can be assumed 
regardless of wording. 



9) Revision required 

(1) + scan decision to improve scan specification, or 

to scan for additional items and ideas. 

(2) M-Group answers which it is expected might be 

better handled through motivational 
clustering. (See pages 237-238) 



3 . Initially we tried to distinguish between sense from the program 
viewpoint and sense from the student's viewpoint. This raised 
problems of assigning credit for fortuitous first-pass performance, 
and in the assessment of partial relevance, so it was decided to 
combine the two on an empirical basis. We consider that if the 
frame performs meaningfully for the student it is immaterial 
whether or not the operation was planned in exactly the way 
it takes place, and whether or not all of the expository material 
one might have liked to call forth in a given situation actually 
appears. Credit is taken for situations where slight discrepancies 
in meaning stand a very good chance of being corrected in the 
next pass. 
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APPENDIX G 



« B-FRAME » FLOWCHART 

1. Attached, 

2. The attached flowchart reflects program changes incorporated 
in the « B-Frame >\ Appendix B notes apply (p. 246-248), except 
as provided below : 

5) (revised) The group of operations to the left of « DUNNO » is devoted 
to the provision of information intended to help the student solve the 
problem. If no specific information need is detected, his response is 
tested for motivational indicators (M2, Mi, & Mo) in three separate 
scans whose specifications are given in Appendix H. The Q-return 
display for each of these differs from the other two in the way it ap- 
proaches the problem. The general «I don’t know » + GARB, and 
« I don’t know » scans are retained in order to catch all such answers 
that escape earlier match. 

9) (new) Satellites are increased from five to six in number, and a slight 
structural modification is provided : Instead of a single block of text 
ending with « Does this explain the term adequately for you ? » a 
diiferendy worded ending is used for students who have been assessed 
« Mi » and are first being reinforced on something they may have some 
knowledge of: « I hope this confirms your understanding of ... (its ■: ... 

What would you like to follow up next ? » 




B-Frame Flowchart 
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B -Frame Flowchart (Continuation) 
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APPENDIX H 



The « B-Frame » program 

1. Attached. 

2. The notes to Appendix C apply (p. 248-249) except as provided 
below : 

8 ) ( omitted ) 

9) (new) A word partially underlined, in a scan specification, means that 
only the underlined portion is stipulated as a match requirement. The 
characters which are not underlined are included only to show the 
reader what term group the programmer has in mind. For example 
« STORAGE » will accept « store », « stored », « storing », and « sto- 
rage » but not «stop». It will also accept the name of the bird, but the 
programmer is willing to take the risk. 

3. It should not be imagined that a CAI programmer would 
often invest in a single frame the amount of work which has 
clearly gone into both the A-Frame and the B-Frame. As a stand- 
alone unit we had to provide it with features which normally 
would need to be written only once fDr an entire course, e.g., 
the « POSIT » and « NEGAT » expansions, and the M-series 
of scan specifications. Also it should be mentioned that the 
command statements can be put together quite rapidly after one 
has worked with a language for a while. As a matter of record, 
one person designed and wrote the « A-Frame » in a little over 
four hours. 
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Reaction text displays — « B -Frame » 

1. List attached. 

2. Appendix D comments (p. 259) apply, with the exception 
of para. 6 (sample dialogues omitted). 



I. First pass 

Type of answer (numbers and letters Program display reaction 
establish categories referred to in («E» = «exit to next frame ») 
Appendix K). 



1. Right — both elements 

a. Reason : ease of distribution, of 
mailing, of sending. 



b. Reason : cost, cheap, 
pensive, save money. 



c. Reason : i.a + i.b 



You are correct in picking 
microfiche, and ease of distribu- 
tion and use is one of its com- 
pelling advantages. Microfiche 
can be duplicated very readily 
and inexpensively... (etc.) ... 
(same as corresponding 
« FICHUP » text in Appendix 
D) (E) 

inex- You are correct in picking 
microfiche, and cost considera- 
tions certainly enter into the 
picture. Microfiche can be du- 
plicated ... (etc., same as 
above). (E) 

Good. You correctly picked 
two of the important advan- 
tages of fiche. Microfiche can 
be duplicated... (etc.;- same as 
above). (E) 



2. Right — first element only 

a. No reason given Why ? 

b. Wrong reason given 

(1) experience (« because had Can you compare them more 
more experience with it »). on the basis of information ? If 

you would like to know more 
about the other forms, or 
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(2) capacity 



(3) clarity 

(4) knowledge (« the only one 
I know about ») 

(5) (n) (other aspects provided 
in the program) 



3. Wrong — Two first elements 



4. Wrong — Wrong first element 

(a) Microcard 

(b) Aperture card 

5. Wrong — Don’t know 

a. « don’t know » with no addi- 
tional element following, and 
no match on motivational in- 
dicators. 

6. Wrong — Don’t know + (un- 

identified matter) 

a. « don’t know » with added 
material, but no match on 
motivational indicators 

b. Mo 
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about the general requirement 
for dissemination of technical 
literature, just type « Clarify 
». 

You are correct in picking 
microfiche, but the reason it is 
superior for dissemination of 
technical literature is not that it 
has greater capacity (which 
depends on the reduction ratio 
in use) but on the iact that 
microfiche can be duplicated 
... (etc.) (E) 

(similar to 2.b.2 above) (E) 
(similar to 2.b.2 above) (E) 

(similar to 2.b.2 above, except 
that some will exit from the 
frame while others will require 
recursion). 

See if you can pick just one, on 
the basis of what you can learn 
from me on the subject. 



Why? 

Why? 



I’ll be glad to explain any of the 
terms I’ve used, if you wish. 
Just type « Clarify ». 



I don’t understand exactly 
what you need to have clarified. 

I realize it is almost impossible 
for you to answer without 
knowing something about the 
types of microform mentioned 
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and about the factors involved 
in dissemination of technical 
literature. I’d like to help von 
tackle the problem from the 
beginning by supplying all the 
information you need. Are you 
willing to work with me toward 
a correct solution ? 

c. Mi Well, suppose we start by 

examining the items you may 
already have beard about > I may 
be able to add some useful 
information about them. How 
about microfiche ? Microcard ? 
Aperture card ? The dissemina- 
tion of technical literature ? 
Are you acquainted with any 
of these ? 

7. Wrong — Don’t know + single 
identified 

a. Unit, unitized 



b. Transparent, film, sheet, fiche, 
microfiche 

c. Opaque, microcard 

d. Aperture 

e. Dissemination, technical 
literature 

8. Wrong — Don’t know + multiple 
identified 

a. Any combination of 7-a, 7*b, I understand you need more in- 
7.C, 7.d, 7.e formation about (7.a, 7.b, 7.C, 

7.d, 7.e). We will take them up 
in that order. (Stacked sub- 
routines of 7.a-.e) 

9. Wrong — Don’t know + misc- 

ellaneous 



I understand you need more in- 
formation about unit, or uni- 
tized, microforms. A unit mi- 
croform is one which... (etc., 
same as corresponding text in 
Appendix D) 

(similar to 7. a) 

(similar to 7. a) 

(similar to 7. a) 

(similar to 7*a) 
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a. Any, all 



b. No experience 



c. Positive, negative 



One of them is superior in 
reproduceability, cost, general 
use, etc. Do you need more 
information ? If so, type « Clar- 
ify » for explanation 

of individual elements. 

I*d like you to choose not on 
the basis of experience but on 
the basis of available informa- 
tion, if possible. Type « Clar- 
ify » for explanation 

of individual elements. 

Microcard is always distributed 
as a positive; microfiches and 
aperture cards may be had in 
either positive or negative 
versions. Aside from this con- 
sideration, what would be your 
choice ? 



d. Not competent, ill equipped, I’ll be glad to explain any of the 
not prepared, not equipped terms I’ve used, if you wish. 

Just type « Clarify ». 



10. Wrong — Rejection 
a. Dislike 



b. Expletive. 



Wrong — 
Clarify, 



a. 



Request 
define, info, 



Aside from your personal feel- 
ings in the matter, can you 
express an opinion as to which 
might be least ////suitable for the 
stated purpose ? 

(same as corresponding text in 
Appendix D. No evidence yet 
as to how this might work). 



describe (same as 7 and 8. The variation 
between request words for Mo 
and Mi is provided to facilitate 
channeling of operations called 
forth by these terms, and for 
scoring if and when incor- 
porated). 

How's that again ? I don't 
understand your response as 



7 



12. Fail 
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worded. Please explain 
fully. 

II. Second pass 

x 3 * Right — Both elements (not applicable. (Frame 

pleted on first pass)). 

14. Right Second Element, first 
element right 

a. Reason : ease of distribution, of (san e as i.a) (E) 
mailing, of sending 

b. Reason : cost, cheap, inex- (same as i.b) (E) 
pensive, save money 

c. Reason : i.a + i.b (same as i.c) (E) 

1 5 . Right Second Element first element (same as corresponding 

wrong tions in Appendix D) 

16. Wrong Second Element, first ele- (ditto) 
ment right. 

17. Wrong — Don’t know Repeats 5. 

18. Wrong — Don’t know — Un- Repeats 6. 
identified 

18. a. Wrong — Don’t know + single Similar to 7, 

identified 

19. Wrong — Don’t know + multiple Similar to 8. 
identified 

20. Wrong — Don’t know + misc- Repeats 9. 
ellaneous 

21. Wrong — Rejection 

a. Dislike Repeats io.a 

22. Wrong — Request (same as 7, 8 and 11) 
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Subroutine following 7 



23. Right 




a. « yes » 


Fine, If you are in doubt about 
any of the other terms, don’t 
hesitate to ask. Here is the original 
question again. (Repeats question). 


24. Wrong 




a. a. « no » 


Rather than spend more CAI 
time on it now, I suggest that you 
either sign off and seek out an 
instructor, or type « PASS » and 
I will go on to the next question. 

(E) 



Subroutine following 8 



25. Right 




a. « yes » 


In that case we will pass to the 
meaning of (next occurring item 
of requested information). 




26. Wrong 




a. « no » 


(same as 24) 



Subroutine following 6b (Handled within main frame) 



27. Right 


a. « yes » alone 


Which one would you like me to 
describe first ? Type « Describe 


b. « yes » with item, 


I understand you need more in- 


or item alone 


formation about... (item)... (etc.) 


28. Wrong 


a. « no )) 


Perhaps you should sign off, then. 
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Subroutine following 6c (Handled within main frame) 



29. Right 




a. « yes » alone 


Which? 


b. « yes » with item, 
or item alone. 


« DUNNO » is automatically in- 
serted in response, so that pro- 
gram goes direct to satellite 


30. Wrong. 




a. « no » 


Shall we take them up one at a 
time ? Suppose we start with the 
problem of dissemination of tech- 
nical literature. Following that, 
you can ask for other elements of 
the problem. 


31. FAIL — If unused on 1st 


pass (same as 12, above) 


32. FAIL — Second instance 


Admit frame failure. 


r . Third pass 


(Predictable only if a subroutine 
is used on the 1st or 2nd pass). 


Subroutine following 8 and 9 


33. Wrong — continue 




a. « PASS » 


All right. But please get clari- 
fication at the first opportunity. 

(E) 


34, Wrong — Exit 




a. « I quit » 


You are now signed off. (E) 



/ 35. FAIL — If ^31 and #32 have 

l been used, program falls through 

l to the next frame. 
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APPENDIX J 

« B » Student answers and program responses 

This Appendix lists the student answers of the « B » set, 
together with the reactive text which would have been called 
forth in the « B-Frame », 





STUDENT INPUT 


REACTION TEXT DISPLAY 


# 1 


I don’t even know what a micro- 
fiche is. 


I understand you need more informa- 
tion about microfiche - (test of 
MICROZ satellite) - Does this explain 
the term adequately for you ? 


# 2 


I don’t know. 


I realize it is almost impossible for you 
to answer without knowing something 
about the types of microforms men- 
tioned and about the factors involved 
in dissemination of technical literature. 
I’d like to help you tackle the problem 
from the beginning by supplying all 
the information you need. Are you 
willing to work with me toward a 
correct solution ? 


# 3 


I am not familiar with programming. 


I don’t understand what you need to 
have clarified. 


# 4 


I have not had any experience with 
any, so cannot decide. 


Can you compare them not on the 
basis of experience but on the basis of 
information? If you would like to 
know more about the general require- 
ments for dissemination of technical 
literature, just type « Clarify ». 


# 5 


I am not sufficiently acquainted with 
the characteristics of these different 
microforms to answer. 


Well, suppose we start by examining 
the items you may already have heard 
about. I may be able to add some 
useful information about them. How 
about microfiche ? Microcard ? Aper- 
ture card ? The dissemination of 
technical literature ? Are you acquaint- 
ed with any of these ? 
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STUDENT INPUT 


REACTION TEXT DISPLAY ! 


# 6 


Lack of knowledge prevents a con- 
sidered answer. 


(Same as #5) 


# 7 


What’s an aperture card. To tell the 
truth, 1 strongly dislike microforms 
and readers. 


I understand you need more informa- 
tion about aperture cards. - (text of 
APERZ satellite) - Does this explain 
the term adequately for you ? 


# 8 


Microfiche, since it is easy to handle, 
store, and reproduce. 


You are correct in picking microfiche, 
and ease of storing is certainly an ad- 
vantage, but one which is also shared 
by the other two forms. Microfiche can 
be duplicated very readily and in- 
expensively, and in fact is the standard 
used by the Federal government for 
distribution of Clearinghouse, DOD, 
NASA, AEC, and ERIC documents, 
which include a large portion of our 
technical literature output... (etc.) 


# 9 


I don’t understand all the distinctions 
among the three types. 


One of them is superior in repro- 
duceability, cost, general use, etc. Do 
you need more information ? If so, 
type « Clarify ». 


#10 


1. Because I don’t quite know what the 
other two are l 


See if you can pick just one, on the 
basis of what you can learn from me 
on the subject. 


#11 


1, as it contains more information 
(data) readily & quickly available. 


You are correct in picking microfiche, 
but the reason it is superior for the 
dissemination of technical literature is 
not that it has greater capacity (which 
depends on the reduction ratio in use) 
but on the fact that microfiche can be 
duplicated... (etc.) (same as #8) 


#12 


I want aperture cards. Aperture cards 
are fun. 


Why? 
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STUDENT INPUT 


REACTION TEXT DISPLAY 


#13 


1 - easy storage 


(same as #8) 


#14 


Fiche. Equipment feasibility, cost, and 
eventual wide availability. 


You are correct in picking microfiche, 
and cost considerations are an impor- 
tant factor. Microfiche can be dupli- 
cated very readily and inexpensively... 
(etc.) (same as #8) 


#15 


Type 1. Largest amount of experience 
with this type. 


You are correct in picking microfiche. 
Can you compare them not on the 
basis of experience but on the basis of 
information? (etc.) (same as #4) 


#16 


It depends on the dissemination system 
and the ultimate user(s), but 1 seems 
best. 


Why? 


#17 


Microfiche, because of Federal stand- 
ard, cost, and availability of readers. 


(same as #14) 


#18 


L Density and handling are superior 
to 3. copying superior to 2. 


(same as #10 FAULT !) 


#19 


Microfiche. Large amount of informa- 
tion stored by unit and « flexible » 
display. 


(same as #11) 


#20 


I have used microfiche with a projector 
and it seems to be quite satisfactory. 
I have not used the other two so have 
no opinion of their usefulness. 


(same as # 10) 


#21 


What is opaque microcard ? 
What is aperture card ? 


I understand you need more informa- 
tion on microcard - aperture card. We 
will take them up in that order. (Text 
of OPAQZ satellite) - Does this ex- 
plain the term adequately for you ? 
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STUDENT INPUT 


REACTION TEXT DISPLAY 


#22 


Don’t know what aperture card is. 
What is microform card with all the 
pages on one card called ? 


I understand you need more informa- 
tion on aperture card - (text of APERZ 
satellite) - Does this explain the term 
adequately for you ? 


#23 


What is an aperture card ? 


\same as #22) 


#24 


I don’t know what a mlcrocard or 
aperture card are. I prefer hard copy 
anyway. 


(same as #21) 


#25 


Microfiche, I guess. Access time in 
microfilm is too long I have never 
used opaque microcard. 


See if you can pick just one, on the basis 
of what you can learn from me on the 
subject. 



APPENDIX K 

« B-FrAME » PERFORMANCE SUMMARY 

1. Attached, 

2. For explanation of column headings, see Appendix F (p. 270-271). 
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Abstract 

A stand-alone CAI frame was written around a question designed to elicit 
a wide variety of replies, in order to measure the utility of reply analysis in 
refining individual frames. Half of a real sample of fifty handwritten responses 
were submitted to the frame, with a match score of 96 % and a sense score 
of 48 %. After revision, the frame attained a match score of 100 % and a 
sense score of 88 %, on the other half of the test set. 

The authors, discuss selected-string-match CAI strategies, including the 
use of attitudinal scan combinations to indicate cognitive state. Mechanics of 
the CAI « macroframe » and its « satellites » are detailed. 



R£sum£ 

A propos d’un projet sur l’instruction par les machines k calculer, un seul 
cadre iso \6 fut £crit sur une question precise afin d’obtenir une grande vari£t£ 
de rdponses, et d^tre 4 meme de mesurer la valeur de Panalyse des r^ponses 
fournie par les cadres individuels. La moiti£ d’un vrai module de cinquante 
rdponses dcrites k la main furent soumises au cadre, avec une congruence 
statistique de 96 % et un systdme de comprehension de 48 %. Apr&s avoir 
6 t 6 revise, le cadre atteignit un resultat de 100 %, et un syst£me de comprehen- 
sion de 88 % sur 1 ’autu.e moitie du contr 61 e d’essai. Les auteurs discutent les 
strategies de « selected string match CAI » comprenant l’usage des combinai- 
sons parcourues pour indiquer l’etat des connaissances. On discute les m£ca- 
niques en grand cadre et ses satellites. 



ZUSAMMENFASSUNG 

Fui vine Frage der Programmierten Instruktion wurde ein unabhangiger 
Rahmen geschrieben, der entworfen wurde urn eine Reihe von Resultaten 
zu bekommen und um die Brauchbarkeit der Antwortenanalyse zur Ver- 
feinerung einzelner Rahmen zu messen. Die Halfte der Originalserie von 
fiinfzig handgeschriebenen Antworten wurde in den Rahmen gefuttert und 
erreichte einen Erfolgsgrad von 96 % und einen Verstandnisgrad (von der 
Seiten des Systems) von 48 %. Nach der Revision erzielte der Rahmen mit 
der anderen Halfte der Versuchsserie einen Erfolgsgrad von 100 % unter 
Verstandnisgrad des Systems von 88 %. 

Die Autoren beschreiben die Auswahl-Aufeinanderfolge-Anpassung (se- 
hcted-string-matc})) Strategien des Programmierten Unterrichts, einschliesslich 
die Anwendung von verhaltensskandierenden Kombinationen, um auf das 
Erkennungs stadium hinzuweisen. Die Mechanik des « Makrorahmens » und 
seines « Satelliten » werden beschrieben. 
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